{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xB2cQUShkXNm"
   },
   "source": [
    "# Ungraded Lab: Image Data Preprocessing with a Validation Set\n",
    "\n",
    "In this lab, you will build upon the previous exercise and add a validation set. This will let you measure how well the model performs on data it hasn't seen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l5FfBGV5yUjb"
   },
   "source": [
    "## Import the Libraries\n",
    "\n",
    "Start with importing all the libraries you will need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import numpy as np\n",
    "from io import BytesIO\n",
    "\n",
    "# Plotting and dealing with images\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "# Interactive widgets\n",
    "from ipywidgets import widgets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l5FfBGV5yUjb"
   },
   "source": [
    "## Inspect the Dataset\n",
    "\n",
    "Similar to the previous lab, you will define the directories containing your images. This time, you will include those with validation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NR_M9nWN-K8B"
   },
   "outputs": [],
   "source": [
    "TRAIN_DIR = 'horse-or-human'\n",
    "VAL_DIR = 'validation-horse-or-human'\n",
    "\n",
    "# Directory with training horse pictures\n",
    "train_horse_dir = os.path.join(TRAIN_DIR, 'horses')\n",
    "# Directory with training human pictures\n",
    "train_human_dir = os.path.join(TRAIN_DIR, 'humans')\n",
    "\n",
    "# Directory with validation horse pictures\n",
    "validation_horse_dir = os.path.join(VAL_DIR, 'horses')\n",
    "# Directory with validation human pictures\n",
    "validation_human_dir = os.path.join(VAL_DIR, 'humans')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LuBYtA_Zd8_T"
   },
   "source": [
    "Now see what the filenames look like in these directories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4PIP1rkmeAYS"
   },
   "outputs": [],
   "source": [
    "train_horse_names = os.listdir(train_horse_dir)\n",
    "print(f'TRAIN SET HORSES: {train_horse_names[:10]}')\n",
    "\n",
    "train_human_names = os.listdir(train_human_dir)\n",
    "print(f'TRAIN SET HUMANS: {train_human_names[:10]}')\n",
    "\n",
    "validation_horse_names = os.listdir(validation_horse_dir)\n",
    "print(f'VAL SET HORSES: {validation_horse_names[:10]}')\n",
    "\n",
    "validation_human_names = os.listdir(validation_human_dir)\n",
    "print(f'VAL SET HUMANS: {validation_human_names[:10]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HlqN5KbafhLI"
   },
   "source": [
    "You can find out the total number of horse and human images in the directories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "H4XHh2xSfgie"
   },
   "outputs": [],
   "source": [
    "print(f'total training horse images: {len(os.listdir(train_horse_dir))}')\n",
    "print(f'total training human images: {len(os.listdir(train_human_dir))}')\n",
    "print(f'total validation horse images: {len(os.listdir(validation_horse_dir))}')\n",
    "print(f'total validation human images: {len(os.listdir(validation_human_dir))}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C3WZABE9eX-8"
   },
   "source": [
    "Now take a look at a few pictures to get a better sense of what they look like. Display a batch of 8 horse and 8 human pictures. You can rerun the cell to see a fresh batch each time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Wpr8GxjOU8in"
   },
   "outputs": [],
   "source": [
    "# Parameters for your graph; you will output images in a 4x4 configuration\n",
    "nrows = 4\n",
    "ncols = 4\n",
    "\n",
    "# Set up matplotlib fig, and size it to fit 4x4 pics\n",
    "fig = plt.gcf()\n",
    "fig.set_size_inches(ncols * 3, nrows * 3)\n",
    "\n",
    "next_horse_pix = [os.path.join(train_horse_dir, fname)\n",
    "                for fname in random.sample(train_horse_names, k=8)]\n",
    "next_human_pix = [os.path.join(train_human_dir, fname)\n",
    "                for fname in random.sample(train_human_names, k=8)]\n",
    "\n",
    "for i, img_path in enumerate(next_horse_pix + next_human_pix):\n",
    "    # Set up subplot; subplot indices start at 1\n",
    "    sp = plt.subplot(nrows, ncols, i + 1)\n",
    "    sp.axis('Off') # Don't show axes (or gridlines)\n",
    "\n",
    "    img = mpimg.imread(img_path)\n",
    "    plt.imshow(img)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5oqBkNBJmtUv"
   },
   "source": [
    "## Building a Small Model from Scratch\n",
    "\n",
    "You will define the same model architecture as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qvfZg3LQbD-5"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential([\n",
    "    # Note the input shape is the desired size of the image 300x300 with 3 bytes color\n",
    "    # This is the first convolution\n",
    "    tf.keras.Input(shape=(300, 300, 3)),\n",
    "    tf.keras.layers.Conv2D(16, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2, 2),\n",
    "    # The second convolution\n",
    "    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2,2),\n",
    "    # The third convolution\n",
    "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2,2),\n",
    "    # The fourth convolution\n",
    "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2,2),\n",
    "    # The fifth convolution\n",
    "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2,2),\n",
    "    # Flatten the results to feed into a DNN\n",
    "    tf.keras.layers.Flatten(),\n",
    "    # 512 neuron hidden layer\n",
    "    tf.keras.layers.Dense(512, activation='relu'),\n",
    "    # Only 1 output neuron. It will contain a value from 0 to 1 where 0 is for 'horses' and 1 for 'humans'\n",
    "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s9EaFDP5srBa"
   },
   "source": [
    "You can review the network architecture and the output shapes with `model.summary()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7ZKj8392nbgP"
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PEkKSpZlvJXA"
   },
   "source": [
    "You will also use the same compile settings as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8DHWhFP_uhq3"
   },
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer=tf.keras.optimizers.RMSprop(learning_rate=0.001),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sn9m9D3UimHM"
   },
   "source": [
    "### Data Preprocessing\n",
    "\n",
    "Now you will setup the datasets with the `tf.data` API. It will mostly be the same as last time but notice the additional code to also prepare the validation data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ClebU9NJg99G"
   },
   "outputs": [],
   "source": [
    "# Instantiate the training dataset\n",
    "train_dataset = tf.keras.utils.image_dataset_from_directory(\n",
    "    TRAIN_DIR,\n",
    "    image_size=(300, 300),\n",
    "    batch_size=32,\n",
    "    label_mode='binary'\n",
    "    )\n",
    "\n",
    "# Instantiate the validation set\n",
    "validation_dataset = tf.keras.utils.image_dataset_from_directory(\n",
    "    VAL_DIR,\n",
    "    image_size=(300, 300),\n",
    "    batch_size=32,\n",
    "    label_mode='binary'\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will scale both datasets to have `[0,1]` range of pixel values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hP_TERHNIJa1"
   },
   "outputs": [],
   "source": [
    "# Define the rescaling layer\n",
    "rescale_layer = tf.keras.layers.Rescaling(1./255)\n",
    "\n",
    "# Rescale both datasets\n",
    "train_dataset_scaled = train_dataset.map(lambda image, label: (rescale_layer(image), label))\n",
    "validation_dataset_scaled = validation_dataset.map(lambda image, label: (rescale_layer(image), label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, you will call a few more methods to configure and optimize your datasets for training. Notice that you don't need to shuffle the validation set. That can be skipped because you are only using the validation set to evaluate the performance of your model. It will not affect how it updates its parameters during training. That is determined purely from the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vQitxc87IZiQ"
   },
   "outputs": [],
   "source": [
    "SHUFFLE_BUFFER_SIZE = 1000\n",
    "PREFETCH_BUFFER_SIZE = tf.data.AUTOTUNE\n",
    "\n",
    "train_dataset_final = (train_dataset_scaled\n",
    "                       .cache()\n",
    "                       .shuffle(SHUFFLE_BUFFER_SIZE)\n",
    "                       .prefetch(PREFETCH_BUFFER_SIZE)\n",
    "                      )\n",
    "\n",
    "# Configure the validation dataset\n",
    "validation_dataset_final = (validation_dataset_scaled\n",
    "                            .cache()\n",
    "                            .prefetch(PREFETCH_BUFFER_SIZE)\n",
    "                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mu3Jdwkjwax4"
   },
   "source": [
    "### Training\n",
    "Now train the model for 15 epochs. Here, you will pass parameters for `validation_data` and `validation_steps`. With these, you will notice additional outputs in the print statements: `val_loss` and `val_accuracy`. Notice that as you train with more epochs, your training accuracy might go up but your validation accuracy goes down. This can be a sign of overfitting and you need to prevent your model from reaching this point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Fb1_lgobv81m",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "history = model.fit(\n",
    "      train_dataset_final,\n",
    "      epochs=15,\n",
    "      validation_data = validation_dataset_final,\n",
    "      verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have a validation set you can plot the accuracy of the training vs validation set over the course of the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the training and validation accuracies for each epoch\n",
    "\n",
    "acc = history.history['accuracy']\n",
    "val_acc = history.history['val_accuracy']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(acc))\n",
    "\n",
    "plt.plot(epochs, acc, 'r', label='Training accuracy')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation accuracy')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend(loc=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When training accuracy outperforms validation accuracy this is usually a sign that your model is overfitting. You will learn more about what to do about this in Course 2!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o6vSHzPR2ghH"
   },
   "source": [
    "### Model Prediction\n",
    "\n",
    "Now take a look at actually running a prediction using the model. This code will allow you to choose 1 or more files from your file system, upload them, and run them through the model, giving an indication of whether the object is a horse or a human."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DoWp43WxJDNT"
   },
   "outputs": [],
   "source": [
    "# Create the widget and take care of the display\n",
    "uploader = widgets.FileUpload(accept=\"image/*\", multiple=True)\n",
    "display(uploader)\n",
    "out = widgets.Output()\n",
    "display(out)\n",
    "\n",
    "def file_predict(filename, file, out):\n",
    "    \"\"\" A function for creating the prediction and printing the output.\"\"\"\n",
    "    image = tf.keras.utils.load_img(file, target_size=(300, 300))\n",
    "    image = tf.keras.utils.img_to_array(image)\n",
    "    image = rescale_layer(image)\n",
    "    image = np.expand_dims(image, axis=0)\n",
    "    \n",
    "    prediction = model.predict(image, verbose=0)[0][0]\n",
    "    \n",
    "    with out:\n",
    "        if prediction > 0.5:\n",
    "            print(filename + \" is a human\")\n",
    "        else:\n",
    "            print(filename + \" is a horse\")\n",
    "\n",
    "\n",
    "def on_upload_change(change):\n",
    "    \"\"\" A function for geting files from the widget and running the prediction.\"\"\"\n",
    "    # Get the newly uploaded file(s)\n",
    "    \n",
    "    items = change.new\n",
    "    for item in items: # Loop if there is more than one file uploaded  \n",
    "        file_jpgdata = BytesIO(item.content)\n",
    "        file_predict(item.name, file_jpgdata, out)\n",
    "\n",
    "# Run the interactive widget\n",
    "# Note: it may take a bit after you select the image to upload and process before you see the output.\n",
    "uploader.observe(on_upload_change, names='value')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-8EHQyWGDvWz"
   },
   "source": [
    "### Visualizing Intermediate Representations\n",
    "\n",
    "As before, you can plot how the features are transformed as it goes through each layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-5tES8rXFjux"
   },
   "outputs": [],
   "source": [
    "# Define a new Model that will take an image as input, and will output\n",
    "# intermediate representations for all layers in the previous model after\n",
    "# the first.\n",
    "successive_outputs = [layer.output for layer in model.layers[1:]]\n",
    "visualization_model = tf.keras.models.Model(inputs = model.inputs, outputs = successive_outputs)\n",
    "\n",
    "# Prepare a random input image from the training set.\n",
    "horse_img_files = [os.path.join(train_horse_dir, f) for f in train_horse_names]\n",
    "human_img_files = [os.path.join(train_human_dir, f) for f in train_human_names]\n",
    "img_path = random.choice(horse_img_files + human_img_files)\n",
    "\n",
    "img = tf.keras.utils.load_img(img_path, target_size=(300, 300))  # this is a PIL image\n",
    "x = tf.keras.utils.img_to_array(img)  # Numpy array with shape (300, 300, 3)\n",
    "x = x.reshape((1,) + x.shape)  # Numpy array with shape (1, 300, 300, 3)\n",
    "\n",
    "# Scale by 1/255\n",
    "x = rescale_layer(x)\n",
    "\n",
    "# Run the image through the network, thus obtaining all\n",
    "# intermediate representations for this image.\n",
    "successive_feature_maps = visualization_model.predict(x, verbose=False)\n",
    "\n",
    "# These are the names of the layers, so you can have them as part of the plot\n",
    "layer_names = [layer.name for layer in model.layers[1:]]\n",
    "\n",
    "# Display the representations\n",
    "for layer_name, feature_map in zip(layer_names, successive_feature_maps):\n",
    "    if len(feature_map.shape) == 4:\n",
    "\n",
    "        # Just do this for the conv / maxpool layers, not the fully-connected layers\n",
    "        n_features = feature_map.shape[-1]  # number of features in feature map\n",
    "\n",
    "        # The feature map has shape (1, size, size, n_features)\n",
    "        size = feature_map.shape[1]\n",
    "\n",
    "        # Tile the images in this matrix\n",
    "        display_grid = np.zeros((size, size * n_features))\n",
    "        for i in range(n_features):\n",
    "            x = feature_map[0, :, :, i]\n",
    "            x -= x.mean()\n",
    "            x /= x.std()\n",
    "            x *= 64\n",
    "            x += 128\n",
    "            x = np.clip(x, 0, 255).astype('uint8')\n",
    "\n",
    "            # Tile each filter into this big horizontal grid\n",
    "            display_grid[:, i * size : (i + 1) * size] = x\n",
    "\n",
    "        # Display the grid\n",
    "        scale = 20. / n_features\n",
    "        plt.figure(figsize=(scale * n_features, scale))\n",
    "        plt.title(layer_name)\n",
    "        plt.grid(False)\n",
    "        plt.imshow(display_grid, aspect='auto', cmap='viridis')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrap Up\n",
    "\n",
    "Great job completing this lab! In the next one, you will see how compacting images can affect your results. As before, please run the cell below to free up resources for the next lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shutdown the kernel to free up resources. \n",
    "# Note: You can expect a pop-up when you run this cell. You can safely ignore that and just press `Ok`.\n",
    "\n",
    "from IPython import get_ipython\n",
    "\n",
    "k = get_ipython().kernel\n",
    "\n",
    "k.do_shutdown(restart=False)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
